1. Field of the Invention
The present invention relates to a method for generating a stereoscopic image and an apparatus therefor, and more particularly, to a method for generating a stereoscopic image efficiently and an apparatus therefor, in which the stereoscopic image having a natural and stable stereoscopic sense which is adaptive to a variety of motion types of a picture is generated and a motion vector of each of I, P and B pictures is extracted and generated from a MPEG (Moving Picture Experts Group) bitstream.
Although a stereoscopic camera adopting two cameras is not used, the present invention enables two-dimensional motion images corresponding to MPEG-1 CD (compact disc), MPEG-2 HDTV (high definition TV) or MPEG-2 DVD (digital versatile disc) to be viewed directly as stereoscopic images. Furthermore, commercial applications of the present invention cover all fields that enable video pictures to be viewed as stereoscopic images. In more particular, the present invention can be applied to analog type video signal fields such as NTSC, PAL and SECAM broadcasting systems as well as all fields using MPEG data.
2. Description of the Related Art
As one prior art technology compared with the present invention, a modified time difference (hereinafter referred to as MTD) method has been developed by SANYO DENKI Co., Ltd., in Japan. Also, a method of converting a vertical parallax into a horizontal parallax has been developed by SAMSUNG ELECTRONICS Co., Ltd., in Korea. These prior art technologies generate stereoscopic images from analog type image signals. The MTD method developed by SANYO DENKI Co., Ltd. 1) extracts a moving area, 2) determines a velocity and direction of motion, 3) determining a delay direction and a delay time from a previous frame based on the velocity and direction of motion, and 4) determining whether a delay image delayed according to the determined delay direction and delay time is viewed through which eye of the right and left eyes.
The method of converting a vertical parallax into a horizontal parallax developed by SAMSUNG ELECTRONICS Co., Ltd., 1) predicts a motion block by block, 2) calculates a magnitude of a block motion vector, 3) converts the magnitude value obtained by 2) into a horizontal parallax value in order to convert all vertical components into horizontal parallax values, 4) moves each block by the horizontal parallax values in the horizontal direction, and 5) generates a mixed picture by gathering the horizontally moved blocks.
However, since one of the previous K frames is selected as a delay image forcedly by motion information in the case of the above-described MTD method, different depths possessed by areas in one frame are neglected. As a result, since a sense of depth does not exist when a stereoscopic image is viewed, a stereoscopic sense exists with respect to a moving object but does not exist with respect to a portion having less movement, for example, a background of a picture.
In the case of the method proposed by SAMSUNG ELECTRONICS Co., Ltd., a distortion of an image occurs due to a horizontal movement of each block in accordance with a difference of a horizontal parallax between neighboring blocks. Accordingly, a natural and stable stereoscopic image cannot be generated adaptively to a variety of motion types of an image.
Thus, in the case of the stereoscopic images generated by the above-described existing methods, a stereoscopic sense cannot be obtained in particular motion types of an image, or a motion of the stereoscopic image is not natural.
In addition, the above-described existing methods should perform a motion prediction method again with respect to a decoded image when a stereoscopic image is generated using MPEG encoded data. Accordingly, motion information contained in a MPEG bitstream is not used. As a result, algorithm and hardware for generating steroscopic images becomes complicated due to the reuse of the motion prediction method.
To solve the above problems, it is an object of the present invention to provide a method for generating an optimal stereoscopic image having a natural and stable stereoscopic sense even in a variety of motion types of an image.
It is another object of the present invention to provide an apparatus for generating an optimal stereoscopic image having a natural and stable stereoscopic sense even in a variety of motion types of an image.
It is still another object of the present invention to provide a method for efficiently generating an optimal stereoscopic image having a natural and stable stereoscopic sense even in a variety of motion types of an image, by using encoded MPEG data.
It is yet another object of the present invention to provide an apparatus for efficiently generating an optimal stereoscopic image having a natural and stable stereoscopic sense even in a variety of motion types of an image, by using encoded MPEG data.
To accomplish the object of the present invention, there is provided a stereoscopic image generation method for generating a stereoscopic image from image data, the method comprising the steps of: (a) determining a motion type of a current picture into one of a still picture, a non-horizontal motion picture, a horizontal motion picture and a fast motion picture, using motion vectors in each picture according to the image data; (b) generating a mixed picture corresponding to the current picture using previous pictures according to the image data and the determined motion type and outputting the current picture and the mixed picture; and (c) receiving the output current picture and the mixed picture and determining the current picture into one of the left picture and the right picture forming a stereoscopic image and the mixed picture into the other thereof, in accordance with the motion type determined in step (a).
To accomplish another object of the present invention, there is also provided a stereoscopic image generation apparatus for generating a stereoscopic image from image data, the apparatus comprising: a motion type determiner for determining a motion type of a current picture into one of a still picture, a non-horizontal motion picture, a horizontal motion picture and a fast motion picture, using motion vectors in each picture according to the image data; a stereoscopic image generator for generating a mixed picture corresponding to the current picture using previous pictures according to the image data and the determined motion type and outputting the current picture and the mixed picture; and a left/right picture determiner for receiving the current picture and mixed picture output from the stereoscopic image generator and determining the current picture into one of the left picture and the right picture forming a stereoscopic image and the mixed picture into the other thereof, in accordance with the motion type determined in the motion type determiner.
To accomplish still another object of the present invention, there is also provided a stereoscopic image generation method for generating a stereoscopic image using MPEG encoded data containing I, B and P pictures, the method comprising the steps of: (a) receiving the encoded picture data and generating and storing motion vectors corresponding to macroblocks in the picture; (b) storing pictures reproducted from the encoded picture data by the number K of pictures; (c) determining a motion type corresponding to a current picture using the motion vectors stored in step (a); (d) generating a mixed picture corresponding to the current picture using the previous pictures stored in step (b) and the motion type determined in step (c) and outputting the current picture and the mixed picture; and (e) receiving the current picture and the mixed picture output from step (d) and determining the current picture into one of the left picture and the right picture forming a stereoscopic image and the mixed picture into the other thereof.
To accomplish yet another object of the present invention, there is also provided a stereoscopic image generation apparatus for generating a stereoscopic image using MPEG encoded data containing I, B and P pictures, the apparatus comprising: a motion vector generation and storing unit for receiving the encoded picture data and generating and storing motion vectors corresponding to macroblocks in the picture; a picture storing unit for storing pictures reproducted from the encoded picture data by the number K of pictures; a motion type determiner for determining a motion type corresponding to a current picture using the motion vectors stored in the motion vector generation and storing unit; a stereoscopic image generator for generating a mixed picture corresponding to the current picture using the previous pictures stored in the picture storing unit and the motion type determined in the motion type determiner and outputting the current picture and the mixed picture; and a left/right picture determiner for receiving the current picture and the mixed picture output from the stereoscopic image generator and determining the current picture into one of the left picture and the right picture forming a stereoscopic image and the mixed picture into the other thereof.
Preferably, the motion vector generation step and the motion vector generator analyzes a header in a GOP (group of picture) of a MPEG encoded bitstream, to extract I, B and P pictures and extracts and generate motion vectors according to macroblocks of each picture to form a motion vector field.
Here, it is preferable that the motion vectors of the B picture and the P picture are extracted, interpolated and generated by macroblocks from the MPEG bitstream corresponding to a corresponding picture, and the motion vectors of the I picture intra-coded are generated using the motion vectors according to the macroblocks of the previous B pictures.
Preferably, the motion type determining step and the motion type determiner analyzes the motion vector field of the current picture, to determine the motion type of the current picture into one of a still picture having no movement, a non-horizontal motion picture in which a camera and/or an object moves in the non-horizontal direction, a horizontal motion picture in which a camera and/or an object moves in the horizontal direction, and a fast motion picture having fast movement.
Here, it is preferable that the step of determining the motion type of the picture into one of the still picture and the motion picture, analyzes the motion vector field of the picture and determines the motion type on the basis of a ratio occupied by the still macroblocks whose horizontal motion component and vertical motion component are all zeroes among the entire macroblocks of the picture.
Here, it is preferable that the step of determining the motion type of the picture into one of the non-horizontal motion picture and the horizontal motion picture, analyzes the motion vector field of the picture and determines the motion type on the basis of a ratio occupied by non-horizontal macroblocks based on a maximum vertical fusion threshold angle and a maximum horizontal parallax threshold value among the entire macroblocks of the picture.
Here, it is preferable that the step of determining the motion type of the picture into one of the fast motion picture and the horizontal motion picture, analyzes the motion vector field of the picture and determines the motion type on the basis of a ratio occupied by fast motion macroblocks based on a maximum horizontal parallax threshold value among the entire macroblocks of the picture.
Preferably, the stereoscopic image generator comprises at least one of a mixed still picture generator for generating a stereoscopic image of the current picture determined into the still picture, a mixed non-horizontal motion picture generator for generating a stereoscopic image of the current picture determined into the non-horizontal motion picture, a mixed fast motion picture generator for generating a stereoscopic image of the current picture determined into the fast motion picture, and a mixed horizontal motion picture generator for generating a steroscopic image of the current picture determined into the horizontal motion picture.
Here, the stereoscopic image generating step corresponding to the current picture determined as a still picture allocates depth information differently from each other based on a brightness of predetermined each block in the still picture and/or the standard deviation thereof, converts the allocated depth information into a horizontal parallax, and generates a mixed picture corresponding to the still picture.
Here, it is preferable that the step of generating a stereoscopic image corresponding to the current picture determined as a non-horizontal motion picture, allocates each parallax value differently from each other between a macroblock whose direction is same as a camera motion direction and the former whose direction is not same as the latter, to thereby generate a mixed picture, in the case that both the camera and the object move in the non-horizontal motion picture. It is preferable that the step of generating a stereoscopic image corresponding to the current picture determined as a non-horizontal motion picture, allocates each parallax value differently from each other between motion macroblocks and still macroblocks, to thereby generate a mixed picture, in the case that only the object moves in the non-horizontal motion picture. It is preferable that the step of generating a stereoscopic image corresponding to the current picture determined as a non-horizontal motion picture, obtains each parallax value corresponding to each current macroblock by using motion vectors of a peripheral edge macroblock according to an edge feature of the current macroblock and a previous picture macroblock to thereby generate a mixed picture, in the case that only the camera moves in the non-horizontal motion picture.
Here, it is preferable that the step of generating a stereoscopic image corresponding to the current picture determined as a fast motion picture, converts the motion vectors of the fast motion macroblocks in the current picture into a maximum horizontal parallax value, moves the converted result in the horizontal direction, and mixes the horizontally moved result with a just previous picture, to thereby generate a mixed picture corresponding to the fast motion picture.
Here, it is preferable that the step of generating a stereoscopic image corresponding to the current picture determined as a horizontal motion picture, determines a previous picture based on a psychophysics-theory-based maximum horizontal parallax threshold value and a predetermined horizontal motion value in the current picture into a mixed picture corresponding to the horizontal motion picture.
Preferably, the left/right picture determining step and the determiner selects a mode A in which the current picture is determined as a left picture and the previous picture is determined as a right picture or a mode B in which the current picture is determined as a right picture and the previous picture is determined as a left picture, adaptively based on a motion type of a camera and/or an object in the horizontal motion picture, in the case that a motion type of the current picture is determined as a horizontal motion picture, to thereby determining a left picture and a right picture forming a stereoscopic image. In the case that the motion type of the current picture is not a horizontal motion picture, it is preferable that the current picture is determined as a left picture and the mixed picture is determined as a right picture.
Here, it is preferable that the determination of a mode A or a mode B in correspondence to the current picture determined as a horizontal motion picture, bisects the current picture into a first decision area being an edge area and a second decision area being a central area and is based on each horizontal motion vector direction of the each macroblock belonging to each of the first and second decision areas, the number and the distribution of the macroblocks.
Preferably, the stereoscopic image generating method and apparatus according to the present invention redetermines the motion type of the current picture primarily determined at the motion type determiner adaptively based on the each combination of motion type of the previous picture and the motion vectors corresponding to the current picture, in order to remarkably decrease a tremble between the stereoscopic images which are temporally successive, and thus performs generation of the stereoscopic image and/or determination of the left/right picture.